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Abstract 

Tsallis and Renyi entropy measures are two possible different generalizations of the 
Boltzmann-Gibbs entropy (or Shannon's information) but are not generalizations 
of each others. It is however the Sharma-Mittal measure, which was already de- 
fined in 1975 (B.D. Sharma, D.P. Mittal, J.Math.Sci 10, 28) and which received 
attention only recently as an application in statistical mechanics (T.D. Frank & 
A. Daffertshofer, Physica A 285, 351 & T.D. Frank, A.R. Plastino, Eur. Phys. 
J., B 30, 543-549) that provides one possible unification. We will show how this 
generalization that unifies Renyi and Tsallis entropy in a coherent picture natu- 
rally comes into being if the q-formalism of generalized logarithm and exponential 
functions is used, how together with Sharma-Mittaks measure another possible ex- 
tension emerges which however does not obey a pseudo-additive law and lacks of 
other properties relevant for a generalized thermostatistics, and how the relation 
between all these information measures is best understood when described in terms 
of a particular logarithmic Kolmogorov-Nagumo average. 
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1 Introduction 



To gain a unified understanding of the different entropy measures and how they 
relate to each others in the frame of a generalized picture, it is first necessary 
to recall what characterizes "classical" entropies and emphasize some aspects 
which are important for the present paper. 

1.1 The Boltzmann-Gibbs entropy and Shannon's information measure 

As it is well known, given a probability distribution P — {pi}, (i — 1, N), 
with pi representing the probability of the system to be in the i-th microstate, 
the Boltzmann-Gibbs (BG) entropy reads 

N 

Sbg(P) = -k ^Pilogpi, 
i=i 

where k is the Boltzmann constant and TV the total number of possible con- 
figurations. If all states are equi-probable it leads to the famous Boltzmann 
principle S = k log W (N=W). BG entropy is equivalent to Shannon's expres- 
sion if we set k = 1 (as we will do from now on) and use the immaterial base 
b for the logarithm function 

N 

S,s(P) = ~J2Pi l °SbPi ■ 

i=l 

It is common to use the natural base for the BG entropy, while base 2 has the 
advantage to deliver information quantities in bits. 

What characterizes BG and Shannon's measure is additivity of information. 
Given two systems, described by two independent probability distributions A 
and B (i.e. P(A PI B) — P(A)P(B)), using an additive information measure 
means that 

S s (AnB)=S s (A) + S s (B\A), 

with 

Ss(B\A) = Y^Pi(A) Ss(B\A = , 

i 

being the conditional entropy. In this case we are talking about extensive sys- 
tems, i.e. systems where the entropy is given by the sum of all the entropies 
of their parts, as it is customary to do in standard statistical mechanics. The 
unique function which assures additivity is the logarithm. Also in the ax- 
iomatic derivation of Shannon's entropy performed by A.I. Khinchin (J6J) , it is 
the additive property which leads to the appearance of the logarithm function. 
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This is the real reason that stands behind the ubiquitous presence of the log- 
arithm function in information theory, and we can confidently say that every 
modification to it reflects a deviation from the additive law. 



We will from now on use the natural base. Shannon's entropy can be written 
in the form of a " linear" (the arithmetic) mean as 



the elementary information gain associated to an event of probability Pi (in 
information theory it is sometimes called the code length). The quantity ^ is 
also called the surprise (less probable events are considered more "surprising" 
than more probable ones), and we will see that it is this quantity which is 
really measured in one way or another, not — logp^. 

1.2 Tsallis 7 entropy 

Additivity is however not always preserved, especially in nonlinear complex 
systems, e.g. when we have to deal with long range forces, as it is in the 
case of the dynamic evolution of star clusters or in systems with long range 
microscopic memory, in fractal- or multifractal-like and self-organized critical 
systems, etc. We are dealing in this case with non-extensive systems; a case 
which received much attention in the last decade. fl5h 

A generalization of the BG entropy to non- extensive systems is known as 
Tsallis entropy fl4). C. Tsallis noted that if non-extensivity enters into the 
play things are described better by power law distributions, p?, so called q- 
probabilities, i.e. by scaled probabilities where q is a real parameter. This 
introduces the formal possibility not to set rare and common events on the 
same footing, as in BG or Shannon statistics, but it enhances or depresses 
them according to the parameter chosen (in complex systems rare events can 
have dramatic effects on the overall evolution). 

With the introduction of the normalized q-probabilities it became customary 
to define so called escort- or zooming-distribution 




lin 




where we will call the quantity 





q > 0, q e 3?. 
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In this frame Tsallis postulated his now famous generalization of Shannon's 
entropy to non-extensivity (flih: 

s T (P.,q) = u = 1 ^ n = — rE^d-pr 1 )- (i-2) 

1 1 1 L i=l 

For q — > 1, Shannon's measure is recovered, i.e.: St(P, 1) = Ss{P) ■ 
Tsallis entropy extends to a pseudo- additive law 

S T (A n S) = 5 r (A) + 5 r (S|A) + (1 - q)S T (A)S T (B\A) , (1.3) 

with 

<Sr(-B| A) = E 7Ti(A) ^(BlA = A;) . 

i 

Let us introduce the generalized q-logarithm function 

x 1 -" - 1 . . 

\0gqX=— , 1.4 

1 — q 

which, for q — 1, becomes again the common natural logarithm. Its inverse is 
the generalized q- exponential function 

e* = [l + (l-q)x}^, (1.5) 

which becomes the exponential function for q — 1. The importance of the 
q-logarithm is that it satisfies a pseudo-additive law 

logq xy = \og q x + \og q y + (l- q)(hg q x)(\og q y) . (1.6) 

Then Tsallis entropy 1.2 can be written as the q-deformed Shannon entropy 

St(P, q) = \og qPl = f> hg q ( I) = (log, [ \ = 

i=l i=l \PiJ \ V"i//lin 

(1.7) 

with the last term resulting as the q-extension of 1.1. This reflects the non- 
extensive character of the system on the elementary information gains. 

Note also that the classical power laws and the additivity rules for the loga- 
rithm and exponential do no longer hold in this generalized context. Except 
for q — 1, in general log q x a ^ a\og q x, which explains why we keep writing 

throughout this paper Shannon's elementary information gain as (log (j:)). 
instead of — (logpj) lin . Useful for our purposes will be the equality 



lin i 



lill 



e x+v+{l- q )xv = e x e y ( L g) 
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We will see how the q-deformed formalism fits naturally in the mathematical 
descriptions of generalized entropy measures. 



1.3 Renyi's entropy 



Either in the case of BG as for Tsallis entropy, in 1.1 and 1.7, an entropy 
measure is the average S obtained over many elementary information gains 
Ii = = log g (^j associated to the i-th event of probability (if the 

system is extensive, q=l). 

Another possible generalization exists and has become commonplace through- 
out the literature, namely Renyi's measure (flih. A. Renyi maintained a still 
additive measure, as in BG entropy, but considered that another form of av- 
eraging is possible. His starting point was the generalized notion of average 
of A.N. Kolmogorov and M. Nagumo ((0), ffioh ) . who independently showed 
that, in the frame of the Kolmogorov axioms of probability theory, the defini- 
tion of the average must be extended to the quasi- arithmetic or quasi-linear 
mean defined as 

5 = /-* (£>/(!,)) , (1-9) 

where / is a strictly monotone continuous and invertible function, the so called 
Kolmogorov-Nagumo function (KN function). On his side, Renyi showed that 
if we restrict to additive measures then only two possible KN functions exist. 
The first one is the common arithmetic mean and is associated with the KN 
function f(x) = x, and the second is the exponential mean with 

f(x) = Cl b^ x + c 2 , (1.10) 

where q is a real parameter, and C\ and c 2 are two arbitrary constants. 

The exponential mean leads to Renyi's information measure or Renyi's en- 
tropy 

1 N 

S R (P,q) = - log 6 $>?, (1.11) 

with b the logarithm base (we will from now on assume the natural base, b = e, 
for Renyi's entropy either). For q — > 1 Renyi's measure becomes Shannon's 
entropy. 

It should be noted how P. Jizba and T.Arimitsu (0) showed that Renyi's 
measure can be obtained also extending the Shannon-Khinchin axioms to a 
quasi-linear conditional information 

S R (B\A) = r 1 (j2^(A)f(S R (B\A t ))) , (1.12) 
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with / as given in 1.10. 



Therefore Shannon's information measure is an averaged information in the 
ordinary sense, while Renyi's measure represents an exponential mean over 



2 The Sharma-Mittal and Supra- extensive entropy 

2.1 Generalizing with Kolmogorov-Nagumo means 

It is important to understand that Tsallis and Renyi entropies are two different 
generalizations along two different paths. Tsallis generalized to non-extensive 
systems, while Renyi to quasi-linear means. But we can search for an entropy 
which generalizes to non-extensive sets and non-linear means containing Tsal- 
lis and Renyi measures as limiting cases. 

Let us unify the picture of all the entropies considered here through KN av- 
erages (as J.Naudts and M.Czachor did (11), tough by a slightly different 
approach) . 

It is immediate to see from 1.7 and 1.9 how for Tsallis's measure it is the KN 
function 



While, for Renyi's measure, choose in 1.10, c\ = = — c 2 (remember 1.4), 
then the KN function takes the form 




f(x) = x 

which averages over the elementary information gain 



(2.1) 




This led us to write it as 




f(x) = log q e x , 



(2.2) 



which, applied on 
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in 1.9 (/ 1 (x) = loge*) leads us to rewrite 1.11 as 



S R (P, q) = (log - 



\ \ r 1 / / exp 

where, of course, ( . ) cxp = ( Li ) cxp stands for the exponential mean defined by 
the KN function 2.2 over the elementary information Jj. 

But, what Tsallis and Renyi measures have in common is that in both cases 

f{h) = log, (~) • (2.3) 



Then, for a further generalization, the simplest step beyond them would be 
that to generalize 2.1 and 2.2 with 

f(x) = log q e* (2.4) 

and set 



h = log s 



where r, s are new parameters on the generalized exponential and logarithm 
functions. Maintaining constraint 2.3 implies s = r. Then calculatin g 1. 9 
(/ _1 (x) = log r e^), one obtains the Sharma-Mittal information measure (13) 

S SM (P,{<l,r}) =\og r e q Kp ' J = (2.5) 




where ( • ) q _ exp stands for an average defined by the KN function 2.4 and that 
we will call the quasi- exponential mean. 

We can see that for r — > 1 Renyi's measure, and for r — ► q Tsallis measure, 
are recovered as limiting cases. 

We will show in the next section that for two statistical independent systems 
A and B it is easy to check that 

S SM (A n B) = Ssm(A) + S S m(B\A) + (1 - r)S SM (A)SsM(B\A) , 
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i.e. a pseudo-additive law holds as in the case of Tsallis entropy. 

Therefore Sharma-Mittal's measure generalizes Renyi's extensive entropy to 
non-extensivity, characterized by the r-logarithm. It is the parameter r which 
determines the degree of non-extensivity, while q is the deformation param- 
eter of the probability distribution (however, when r — ^ q the two parame- 
ters become intertwined and in Tsallis entropy it is q which measures non- 
extensivity) . 

On information theoretic grounds, B.D. Sharma and D.P. Mittal (13), ad- 
vanced already in 1975 this non-additive measure which shows to have a non- 
extensive character either. But it wasn't until recently ((0), (0), and without 
mentioning it explicitly (fill)) that Sharma-Mittal's measure has been investi- 
gated in statistical mechanics. 



2.2 Generalizing with q-logarithms and q- exponentials 



At this point let us see how by using the q-deformed logarithm and expo- 
nential formalism, one could express in a much more compact form the same 
generalization path. 

First of all recall a well known relationship which exists between Tsallis and 
Renyi entropies, namely 

S R (P, q) = ^~ log [1 + (1 - q) S T (P, q)] . (2.6) 
1 - Q 



Here we can efficiently exploit the generalized logarithm and exponential func- 
tions 1.4 and 1.5, rewriting 2.6 in the more compact form 

S R (P,q)=\oge s q ^\ (2.7) 

from which follows immediately 

S T (P,q) = \og q e s ^. (2.8) 



Looking at the structure of 2.7 and 2.8 we can ask if, given another parameter 
r, the following 



S S m(P, ti,r}) = log 



p S T (P,q) 
r °<y 



1 



1-9 



(2.9) 
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and 



— 

S SE (P, {q, r}) = log, &™ = VlM^m^Zl f (2 . 10 ) 



1-q 



might then be other possible generalizations? 2.9 can be recognized imme- 
diately as Sharma-Mittal's measure 2.5 and can be already accepted as an 
extension. 

2.10 instead needs a closer look. For r — > q it obviously boils down to Renyi's 
entropy. For r — > 1 we obtain Tsallis' measure again. So, from a formal point 
of view it can be regarded as another generalization too. It is however not 
entirely clear what kind of statistics it expresses. Its particular status might be 
best evidenced expressing all the measures in terms of (logarithmic averaged) 
surprise quantities. 

Indeed, notice that we can rewrite the quantity 



i 

1-9 
,1 \ - 





1 



(2.H) 

where we used the logarithmic mean (• ) log defined by the KN function f(x) = 
log q x. Then, from 1.7 and 1.11, and using 2.11, equations 2.7 to 2.10 can be 
rewritten as 

S R (P,q) = \og(-} ; (2.12) 




• < i°g 9 (^) ; ( 2 - 13 ) 



S SM {P,{q,r}) = log r (-\ ; (2.14) 

\^/log„ 



log( p . 

S SE (P,{q,r}) = \og q e r 1 log V (2.15) 

With the q-deformed logarithm and exponential formalism we could easily 
see the generalization path to follow and write all the measures into a more 
compact form (2.7 to 2.10). Moreover this makes it easier to recognize the 
behavior of the limits than in their explicit form (the r.h.s. of 2.9 and 2.10). 
With no or only few passages it is immediate to see how 2.9 reduces to Tsallis 
entropy for r —> q, and for r — > 1 it reduces to Renyi's entropy (without any 
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need to apply Hopital rule, first order approximations or whatever, insert 1.2 
in 1.5). 



The limit for q — > 1 for Sharma-Mittal measure is 

lim Ssm = limlog r e Sr = log r e Ss = log r ( — ) 

^ ^ Wlog 



1 — r 

which Frank and Daffertshofer (Q) used to call the gaussian entropy. 



By the way 



lim Sse = lim log ef R = log e s r s = log e r x * /log = 

q-*l q->l 

= log fl - (1 -r)Y^pi\og pi J . 

But the point is that when compared with 2.12, 2.13, 2.14, measure 2.15 seems 
to stand apart and does not correspond to some quasi-linear mean in the style 
of 1.9. 



2.3 Comparing the supra- extensive entropy with Sharma-Mittal' s entropy 



Let us then focus shortly on the separate nature of 2.15 (or 2.10) and some of 
its properties. 

First of all note that it can be shown how for two statistical independent 
systems A and B, similarly to Tsallis' entropy, the Sharma-Mittal's entropy 
obeys a pseudo-additive law and can be decomposed as in 1.3. It is almost 
immediate to see this by employing the generalized exponential formalism. 
Thanks to 1.3, 1.6, 1.8, starting from the middle term of 2.9 we can write 

S SM (AnB) = \og r e s q ^ AnB ^ = 

= log r e q [S T (A) + S T (B\A) +{l-q) S T (A)S T (B\A)) = log r (e s q ^e s q ^) = 
= log r efW + log r ef <*W + (1 - r) log r ef^ logr jh{B\A) = 

= S SM (A) + S SM (B\A) + (1 - r)S SM (A)S S M(B\A) . (2.16) 
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Proceeding in the same manner with 2.10 leads however not to the same 
decomposition. Because of Renyi's measure additive character one can't go 
further than 

S SE (A nfl) = log g e s r «^ = \og q e r [S R (A) + S R (B\A)} 
with Sr(B\A) as given in 1.12. 

Entropy 2.15 therefore obeys a new form of non-extensivity, we call supra- 
extensivity. 

There are also other aspects which should be mentioned. Let us briefly recall 
the notions of concavity and stability applied to entropy measures. 

Given two probability distributions P = {p\, ...,Pn} and P' = {p[, ■■■,p' N } and 
defining an intermediate distribution P" = {p", ...,p" N } with 

p'; = m + (l-fi)p[; V/iG [0,1], 

S(P) is said to be a concave entropic functional if and only if 

s(n>f,s(p) + (i-^s(n. 

Otherwise, S(P) is said to be convex. Concavity implies thermodynamic stabil- 
ity (e.g. thermal equilibrium between two initial temperatures in BG statistical 
mechanics) . 

Recall also the notion of stability (or experimental robustness, as Tsallis calls 
it, in order to avoid confusion with the previous form of thermodynamic sta- 
bility) which implies that for arbitrary small variations of the probabilities Pi 
a statistical functional remains finite. That is, given a deformation 

\\P-P'\\ = Y1 \Pi~Pil ' 

i 

such that \\p — p'\\ < 5 £ , we obtain stability of S(P) if 

S(P) - S{P') 



A 



<e; V<5 £ >0,Ve>0 



Q 

'-'max 

with S max the maximum value S can attain and for all microstates i = 1, N. 



Lesche claims (J8D that this is a necessary condition for an entropy measure to 
be a physical quantity and showed that, while BG entropy is always stable, 
Renyi's measure is unstable for all q ^ 1. It is also known that BG entropy 
is always concave, while Renyis measure is concave only for a < 1, and can 
be either concave or convex for q > 1. More recently, Abe jlj) showed that 
Tsallis entropy is concave and stable for all positive values of q. It might 
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also be worth mentioning that a physical entropy is not only expected to be 
generically concave and Lesche-stable but should lead also to a finite entropy 
production per unit time. BG and Tsallis entropies share all these properties. 
Renyi entropy shares none. (0) 

So, being an extension of it, it is clear that the properties of concavity and 
stability and finite entropy production per unit time are generically violated 
also in Sharma Mittal's and the supra-extensive entropy. 



Finally, it should also be underlined how Frank and Plastino showed (3) that 
the Sharma-Mittal entropy is the only measure that allows for a pseudo- 
additive decomposition and at the same time gives rise to a thermostatistics 
based on escort mean energy values 



U 



(Ei are the energy levels) admitting of a generalized partition function Z de- 
fined by log r Z SM ■= log r Z SM - PU with 



i 

1-9 




5>n = - > ( 2 - 17 ) 



(Zsm is the partition function which takes U while Z$m takes zero as the 
energy reference, and where f3 is an inverse temperature measure), that leads 
to the usual expressions for the free energy 

F = U - TSsm = ~ log r Z SM 

and the mean energy 

d 

U = —Qplog r Z SM . 

We saw that the new measure we considered here does not allow for a pseudo- 
additive decomposition like 2.16, and therefore it is to expect that the partition 
function describing the free and mean energy cannot have the same structure 
Zsm common to Sharma-Mittal entropy unless, as can be shown (§) applying 

log («) log 

the maximum entropy principle, one substitutes 2.17 with Zse — ° Sq ■ 

Summing up, the supra-extensive entropy, does no longer obey a pseudo- 
additive statistics, if based on escort mean values the partition function must 
take an intrinsically different form than Sharma-Mittals one, but what they 
have in common with Reny's entropy is that, in general, they do not possess 
the property of concavity, Lesche-stability and finite entropy production per 
unit time. 
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3 Conclusion 



We showed how the Sharma-Mittal and a new generalized entropy measure 
both unify Tsallis and Renyi entropies on two different paths in a way that 
appears natural and almost immediate when we make use of the generalized 
q-logarithm an q-exponentials as in 2.7, 2.8, 2.9 and 2.10. We underlined 
how the relationship among all measures becomes particularly clear using 
the logarithmic KN average 2.11 rewriting them as in 2.12, 2.13, 2.14 and 
2.15. This path naturally leads to the supra-extensive entropy which does not 
emerge from the KN means approach alone and does not conform to a pseudo- 
additive law, lacks of concavity, Lesche-stability and finite entropy production. 
However, because the new measure here proposed emerges so naturally as 
another possible extension of Renyi and Tsallis entropy it is therefore worth 
of being mentioned. It is tempting to conclude that, while it might not have 
applications in a generalized thermostatistics, it nevertheless might be of some 
interest in the frame of information theory, cybernetics, control theory, etc. 

Finally we obtained a way of understanding all these entropy measures in a 
unified picture that can be summarized in the following table and diagram. 
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771 i 
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Explicit form 


KN-mean form 


KNi og -mean form 


7 ^ , -£ 

tog g x expg-jorm 
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1-9 

r C1 r\ * ^ rtl 1 

1-9 


, ( los (^ 
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/ / exp 






Sharma-Mittal 


1 — r 

T^[(EiP/)^-l 


(log, ( 


Pi)) 


\— exp 
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Tsallis 


v — r /V n 

1-9 


(log, 


(A) 






log 


log ?e 5 «( p ^) 


Renyi 




(log 
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l 0ge 5T(P,<7) 


BG-Shannon 




(log 


(pi))lin 
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log 
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Shannon (Boltzmann-Gibbs) 
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